Global Convergence of Two-Timescale Actor-Critic for Solving Linear Quadratic Regulator

The actor-critic (AC) reinforcement learning algorithms have been the powerhouse behind many challenging applications. Nevertheless, its convergence is fragile in general. To study instability, existing works mostly consider uncommon double-loop variant or basic models with finite state and action space. We investigate more practical single-sample two-timescale AC for solving canonical linear quadratic regulator (LQR) problem, where actor critic update only once a single sample each iteration on an unbounded continuous Existing analysis cannot conclude such case. develop new framework that allows establishing global to epsilon-optimal solution at most order of epsilon -2.5 complexity. our knowledge, this first finite-time LQR optimality. complexity improves those other variants by orders, which sheds light wisdom algorithms. also further validate theoretical findings via comprehensive simulation comparisons.